The present invention relates to apparatuses, systems, and methods for determining the three-dimensional location of one or more locations on the surface of an object. More specifically, the present invention relates to methods, systems, and apparatuses involving non-contact three-dimensional location estimating using multiple-channel pattern projection and multiple-channel image recording architectures.
Measurement of three-dimensional object surfaces may be needed for a number of applications, including quality insurance for manufacturing, reverse engineering, as-built documentation, facial recognition, machine vision, and medical applications. For many applications, it may be necessary or desirable to measure three-dimensional object profiles without making physical contact with the object under test. One approach to wide-field non-contact three-dimensional surface measurement is stereo vision, whereby a scene or object may be imaged from two or more points of view. With calibrated imagers, it may be possible to triangulate quantitative three-dimensional data describing the scene provided correspondences can be identified between the images. Such correspondences may include singular points, such as corners, or other distinguishable characteristics such as surface texture. For many objects and scenes, however, identifying correspondences may be a significant problem that may impede the use of stereo vision approaches for quantitative three-dimensional measurements. Also, because the quality of the three-dimensional reconstruction depends on recognition of correspondences, stereo vision systems may not provide a guaranteed level of accuracy or resolution that is independent of the scene being imaged.
On the other hand, some wide-field structured illumination approaches to three-dimensional surface measurement solve the correspondence problem by employing a projector capable of illuminating the object with patterned light. These wide-field pattern projection techniques are typically much faster than time-of-flight and laser spot scanning or line scanning approaches. In wide-field active illumination systems, the three-dimensional information describing the surface shape of the object may be encoded in the deformation of the illumination pattern as seen by an imaging camera offset from the projector. Some techniques may solve the correspondence problem by encoding the local angle of the projected illumination at each object location imaged by the camera using one or more patterns and then decoding the projection angles from the intensity data recorded by the camera. These projection angles may then be used, together with the known camera and pattern projector positions and orientations, to reconstruct the surface of the object by triangulation. For robust surface reconstruction, the pattern encoding may need to be faithfully decoded at each imaged location in the presence of optical noise, electronic noise, pattern distortion, object reflectivity and texture variations, illumination and lighting variations, discontinuities and shadowing on the object surface, and other potential sources of error. Robust encoding and decoding of the illumination angle for a wide range of objects and scenes is one of the foremost challenges in the field of three-dimensional surface measurement and profilometry.
The wide variety of wide-field structured illumination three-dimensional surface measurement techniques may be broadly categorized based on the dependence of successful pattern decoding at a given location on the object on the measured values of neighboring locations on the object. Location-dependent, (sometimes referred to as pixel-dependent) algorithms may require a neighborhood of locations to be measured to decode the projection angle and estimate the coordinates at a single location, while location-independent (sometimes referred to as pixel-independent) algorithms may determine the coordinates of each location independently from other locations. While some pixel-dependent approaches may require that the object surface is evenly illuminated, has uniform reflectivity, and contains no abrupt discontinuities due to holes, steps, spikes or shadowed regions, one of the most significant strengths of pixel-independent algorithms is that they may not need to make assumptions about the structure or reflectivity of the object, or about ambient lighting.
Some pixel-independent techniques may use direct encoding of the illumination angle at each location using variations in intensity or variations in wavelength across the illumination pattern, as in the multi-color technique that may be described in U.S. Pat. No. 6,937,348, incorporated herein by reference. Direct encoding techniques, however, may be very sensitive to optical and electronic noise, ambient lighting, detector linearity, as well as object texture, reflectivity and/or coloration. Alternatively (see, for example, Huntley and Saldner, Applied Optics, Vol. 32, 3047-3052, 1993, incorporated herein by reference), pixel-independent temporal phase unwrapping techniques may encode the projection angle in the phases and/or frequencies of time-varying sinusoidal patterns. Such approaches typically rely on projecting moving patterns or multiple stationary patterns and generally require multiple images to be acquired to reconstruct a three-dimensional surface. Some pixel-dependent approaches, such as the sinusoidal fringe phase-shifting method that may be described in U.S. Pat. No. 4,499,492, incorporated herein by reference, may also rely on multiple patterns and images.
Because temporal phase unwrapping approaches may need to project multiple patterns and/or acquire multiple images they may not be suitable for three-dimensional imaging of moving objects or stationary objects in a vibration-prone environment, since object motion relative to the illumination and/or the camera between the projection of successive patterns and acquisition of the corresponding images may corrupt the three-dimensional reconstruction. Furthermore, temporal phase unwrapping techniques may not be suitable for capturing three-dimensional measurements of high-speed single-shot events such as impacts, explosions, and momentary facial expressions that may occur much faster than multiple frames can be acquired. While direct encoding and pixel-dependent techniques may have the potential for high-speed surface profiling, they suffer from object-dependent, lighting-dependent and detector-dependent limitations as described above, and even the highest speed systems typically rely on pattern projection technology based on consumer electronics, such as liquid crystal and digital micromirror projectors, which may limit surface shape measurement speeds to approximately 60 frames per second (see S. Zhang, Optics and Lasers in Engineering, in press, doi:10.10.16/j.optlaseng.2009.03.008, incorporated herein by reference).
One impediment for high-speed pixel-independent three-dimensional shape measurement has been the projection technology. Approaches have utilized a projection paradigm wherein a single, reconfigurable projection channel is used to sequentially project multiple patterns. Such single channel projection techniques include projectors based on interferometers, as may be described in U.S. Pat. No. 6,690,474, incorporated herein by reference; liquid crystal spatial light modulators, as may be described in U.S. Pat. No. 6,208,416, incorporated herein by reference; and digital micromirror devices (see, for example, S. Huang et al., Opt. Eng. Vol. 42:1, 163-168, 2003, incorporated herein by reference). In these approaches, the sequential projection of multiple patterns may limit the speed at which a three-dimensional image can be acquired. On the other hand, a single-channel pattern projection technique using an acousto-optic device (see Mermelstein et al., Optical Engineering Vol. 39, 106-113, 2000, incorporated herein by reference) may be leveraged to project patterns at much faster rates. The speed of this technique, however, may be limited by the speed of acquiring multiple image frames. Although the shift from projection-limited to imaging-limited shape measurement speed is significant, when used with typical commercial cameras, this technique may not offer a large speed advantage over systems based on high-speed pattern projectors.
As a step towards parallelizing high-speed three-dimensional shape measurement, U.S. Pat. No. 6,788,210, incorporated herein by reference, may disclose a method for multiple simultaneous pattern projection and image acquisition using color multiplexing using the red, green, and blue (RGB) pixels of color video projectors and color cameras. Although three frames are typically not sufficient for pixel-independent three-dimensional shape measurement techniques, such RGB multiplexing may be used to implement a pixel-dependent phase-shift method and spatial phase unwrapping using a single frame. RGB multiplexing may also be used to reduce the number of image acquisitions in temporal phase unwrapping three-dimensional shape measurement techniques (see Kinell, Optics and Lasers in Engineering, Vol. 41, 57-71, 2004, incorporated herein by reference). However, the use of RGB pixels may be problematic for imaging of color objects. Furthermore, the use of standard RGB cameras rather than a custom-designed optical multiplexing system may provide only a limited improvement in image acquisition speed, may lead to problems with crosstalk among the channels, and may suffer from detection sensitivity imbalance among the RGB pixels. Moreover, because digital micromirror device (DMD) projectors typically used with RGB-multiplexed imagers generally utilize a single broadband source, a single modulating DMD device, and a spinning color filter wheel to produce patterns with different colors sequentially, the pattern projector may be a fundamentally a single-channel device rather than a parallel optical system and may limit the three-dimensional surface measurement speed regardless of parallelism in the imaging device.
While pixel-independent wide-field three-dimensional surface shape measurement systems may be more robust to variations in object structure and lighting conditions than stereo vision systems, they typically rely on bulky pattern projection systems using high-power illumination sources. On the other hand, emerging portable mobile electronic devices, such as telephones, laptops, PDA's, gaming systems, photographic and video cameras increasingly incorporate powerful processors, displays, and miniature integrated imaging devices and may become the platforms of choice for consumer three-dimensional imaging and object tracking applications that may stand to benefit from the strengths of wide-field structured illumination techniques over stereo vision. However, mobile three-dimensional imaging platforms using active illumination may not only require miniature low-power pattern projectors and cameras, but may also benefit from fast power-efficient data processing algorithms and high-speed pattern projection and data acquisition capabilities to avoid reconstruction artifacts due to object motion and mobile device vibration.
There is thus a need for tools and techniques that can provide robust high-speed, wide-field measurements of the shape of surfaces of three-dimensional objects and scenes for a wide range of objects and under a wide range of lighting conditions. There is also a need for tools and techniques that can measure shapes of surfaces of rapidly moving objects, including single-shot events, and can operate in vibration prone environments. To attain sufficient measurement speeds, such tools and techniques may need to parallelize the processes of pattern projection and image acquisition, instead of using a single sequential reconfigurable pattern generation device and a single sequential imaging system. There is also a need for robust methods and algorithms for measuring locations in a three-dimensional scene and reconstructing the shape of surfaces of three-dimensional objects in presence of optical noise, electronic noise, pattern distortion, object reflectivity and texture variations, illumination and lighting variations, discontinuities and shadowing on the object surface, and other potential sources of error. Furthermore, there is a need for high-speed, miniature, low-power active illumination three-dimensional imaging systems that may be integrated into mobile electronic devices for consumer applications and other three-dimensional surface shape measurement applications including robotics.